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Abstract. Let G be one of the classical Lie groups GL„(M), GL„(C), 0{p,q), 
0„(C), U(p, g), and let G' be respectively the subgroup GL„_i(]R), GL„_i(C), 
0(p, q — 1), 0„_i(C), U(p, q — 1), embedded in G in the standard way. We show 
that every irreducible Harish-Chandra smooth representation of G" occurs with 
multiplicity at most one in every irreducible Harish-Chandra smooth representation 
of G. 
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1. Introduction and main results 
Let G be one of the classical Lie groups 

GL„,(M), GL„,(C), 0{p,q), 0„(C), \J{p,q), 
and let G' be respectively the subgroup 

GL„_i(M), GL„_i(C), 0(p, g - 1), 0„_i(C), U(p, q - 1) 
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of G, where p > 0, q,n > 1. The subgroup G' embeds in G in the usual way as 
follows. For general linear groups, 



GL 



n-l 



a 
1 



G GL„(K) I a E GL„_i(K)| c GL, 



where IK stands for either M or C. The real orthogonal groups are realized as 

where Ip^q is the diagonal matrix of size p + q whose first p diagonal entries are 1, 
and last q diagonal entries are —1. Then 

0(p, g - 1) = G Wi(R) n 0{p, q) C 0(p, q). 

Likewise for the complex orthogonal groups and the unitary groups. 

By a representation of G, we mean a continuous linear action of G on a complete, 
locally convex, Hausdorff, complex topological vector space. We say that a represen- 
tation y of G is a Harish- Chandra smooth representation if it is Frechet, smooth, 
of moderate growth, admissible and Z(0c)-finite. Here and as usual, Z(0c) is the 
center of the universal enveloping algebra U(gc) of the complexified Lie algebra gc 
of G. The reader may consult |C89] and |W92t Chapter 11] for more details about 
Harish- Chandra smooth representations. 

The main purpose of this paper is to prove the following 

Theorem A. Let V and V be irreducible Harish- Chandra smooth representations 
of G and G' , respectively. Then the space of G' -intertwining continuous linear maps 
from V to V is at most one dimensional, i.e., 

dim Home < 1. 

Theorem \K\ and its p-adic analog have been expected (first by Bernstein) since the 
1980's. When V is the trivial representation, Theorem|X]is proved in [AGSlj . |AGS2j 
and |Di], in the case of general linear, orthogonal, and unitary groups, respectively. 
The p-adic analog of Theorem |X] is proved in [AGRSj in its full generality. 



Remark. Denote by K the maximal compact subgroup 

0(n), U(n), 0(p) X 0(g), 0(n), U(p) x U(g) 

ofG, according to the five cases under consideration. Set K' = G'CiK, and denote by 
q'q the complexified Lie algebra of G' . Given Harish- Chandra smooth representations 
V and V of G and G' (respectively), we expect that a certain form of automatic 
continuity theorem will imply that 
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where Vk is the underlying {qq, K)-module of V , similarly for V^i. Therefore, a 
{qc, K)-module version of Theorem\^ should hold. Consequently, we expect the the- 
orem to remain true whenever V and V are irreducible admissible representations . 

For any (smooth) manifold M, denote by (7"°°(M) the space of generahzed func- 
tions on M, which by definition consists of continuous hnear functionals on D^(M), 
the space of (complex) smooth densities on M with compact supports. The latter is 
equipped with the usual inductive smooth topology 

By (a version of) the Gelfand-Kazhdan criterion, Theorem |X] is a consequence of 
the following result. See Proposition 17. 1[ 

Theorem B. Let f e C"^{G) satisfy 

figxg'^) = f{x) 

for all g & G' . Then we have 

fixn = fix), 

where a is the anti-involution of G given by 

{X*, if G is a general linear group, 
x~^, if G is an orthogonal group, 
x~^, if G is a unitary group. 

We record another consequence of Theorem [Bl in the case of general linear groups. 
As before, let K be either M or C. Denote by P„(K) the subgroup of GL„(]K) con- 
sisting of matrices whose last row is [0, 0, ■ ■ ■ , 0, 1]. Since P„(]K) contains GL„_i(]K), 
and since P„(]K), P„(K)* and the center generate the group GL„(K), we have the 
following 

Corollary C. Every generalized function on GL„(]K) which is invariant under the 
adjoint action of Pn(IK) is invariant under the adjoint action of the whole group 
GL„(K). 

We remark that under the additional assumption that the generalized function 
is an eigenvector of the algebra of bi-invariant differential operators on GL„(K), 
Corollary [Cl is the main result of |Ba03j (Theorem 1.4). As observed by Kirillov, 
this implies the validity of his famous conjecture on GL„(K), namely that every 
irreducible unitary representation of GL„(K) remains irreducible when restricted to 
the subgroup P„(K). We refer the readers to |Ba03j for details. 

Here are some words on the approaches, contents and the organization of this 
paper. In Section [21 we examine the space of tempered generalized functions with 
support properties for both the functions and their Fourier transforms, as a module 
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for the Weyl algebra. A key result (Proposition 12. 7p says that certain such modules 
are complete reducible with expected irreducible factors. In Section [3l we introduce 
the notion of a hermitian 74-module, where A is a commutative involutive algebra over 
M. Then the group G in this paper becomes the isometry group U(i?) of a hermitian 
A-module E, corresponding to one of the five simple commutative involutive algebras 
A. We then prove in this context (Proposition 13.31) that a (weighted) Euler vector 
field acts semisimply on a certain space of tempered generalized functions on E, and 
all its eigenvalues are nonnegative integers. Note that the proof of this positivity 
result depends in a rather crucial way the rigidity assertions of Section [2l In Section 
m we introduce a group U(£') and an action of IJ(-E) on \x{E) x E, where u{E) 
is the Lie algebra of U(i?). The group U(i?) has a quadratic character xe with 
kernel U(i?), and the key object of concern is then C^|(u(-E) x E), the space of xe- 
equivariant tempered generalized functions on u{E) x E. We prove in Proposition 
14.11 a reduction result for such generalized functions within the null cone, by using 
metrically properness of nondistinguished nilpotent orbits, or by appealing to the 
eigenvalue estimate of Section [3] for distinguished nilpotent orbits. Sections [21 [3l and 
m are at the heart of our approaches. 

In Section [5l we carry out the reduction to the null cone (Proposition 15.31) by a 
form of Harish- Chandra descent. We then see in Section [6] that results of Sections H] 
and [5] allow us to conclude the vanishing of C^|(u(-E) x E). This leads us to Theorem 
16. 5[ which is a reformulation of Theorem [Bl In Section [TJ we derive Theorem \K\ from 
Theorem [B] by using a version of the Gelfand-Kazhdan criterion. Notwithstanding the 
fact that the general lines of the concluding three sections are known to the experts 
(see |GK75i [Be84l IJR961 lAGRS] for related references), the approaches taken by the 
current article, in terms of hermitian A-modules, have some distinct advantages, at 
least for the problem at hand. 

2. Rigidity of some generalized functions 

Recall the space (7~°°(M) of generalized functions on a manifold M. For any 
locally closed subset Z of M, denote by 

(1) cr'^{M-z)ccr'^{u) 

the subspace consisting of all / which are supported in Z, where U is an open subset 
of M containing Z as a closed subset. This definition is independent of U . 

If M is a Nash manifold, denote by Cr^{M) C Cr^{M) the space of tempered gen- 
eralized functions on M. We refer the interested reader to |Shl lAGl] on generalities 
of Nash manifolds and tempered generalized functions. (For a short introduction, see 
[JSZj .) We say that a subset Z of a Nash manifold M is locally Nash closed if there 
is an open semialgebraic subset U of M, which contains Z as a closed semialgebraic 
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subset. In this case, denote by C~^{M; Z) the subspace of C^^{U) consisting of all 
/ which are supported in Z. Again this is independent of U. 

Let F be a finite dimensional real vector space, which is canonically a Nash man- 
ifold. Denote by W[F] the space of all (complex) polynomial coefficient differential 
operators on F, called the Weyl algebra of F. It contains the algebra C[F] of all 
polynomial functions, and the algebra D[F] of all constant coefficient differential 
operators. Furthermore, the multiplication map 

D[F] (g) C[F] ^ W[F] 

is a vector space isomorphism. 

The space C~^{F) is a VF[F]-module in a natural way. Here is an example of 
irreducible VF[F]-submodule of C~^{F) with a simple structure: C~^{F; {0}). It has 
a distinguished nonzero element 6f (called the Dirac function), which is characterized 
(up to a nonzero scalar) by the equation 

X6f = 0, AG F*, 

where F* is the space of real valued linear functionals on F. More generally, we 
define the following analog of C~^{F; {0}) for each subspace F' of F: 

(2) C^«(F; F'f^' := {/ G Cr^{F; F') \ dv is nilpotent on /, w G F'}. 

Here and henceforth, dv := is the partial derivative along f , and we say that a 
linear operator is nilpotent on a vector if some positive power of the linear operator 
annihilates the vector. 

Lemma 2.1. If F = F' ® F" is a direct sum decomposition, then 

C-^{F; F'f^' = C[F'] ® Cr^{F"; {0}), 
and consequently it is an irreducible W[F]-module. 

Proof. Note that every tempered generalized function has a finite order. Hence by the 
well-known result of L. Schwartz about local representation of a generalized function 
with support, we have 

cr^{F; F') = cr^{F') ® Cr^{F"; {0}). 

The lemma then follows easily. □ 

The following lemma says that Cr^{F; F'f^' is typical within a certain category of 
-modules. It may be viewed as an algebraic version of the Stone- Von Neumann 
theorem. See |W93j . Lemma 3 of Appendix 1. 

Lemma 2.2. Let V he a W[F]-module such that every A G {F/F')* and every dv 
(v G F' ) act locally nilpotently. Then V splits into a direct sum of irreducible W[F]- 
modules, each of which is isomorphic to C~^{F]F'Y^' . 
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Prom now on, we further assume that F is a non-degenerate real quadratic space, 
i.e., it is equipped with a non-degenerate symmetric bihnear form {, )f- Recall that 
the Fourier transform is the topological linear automorphism 

J^F ■■ S(F) S{F) 
of the Schwartz function space S{F), given by 

{J^FDix) = Jj(y)e-^-V^i-'y)- dy. 

Here dy is the Lebesgue measure on F, normalized so that the volume of the cube 

{^it;! + t2V2 H h CrVr | < , ^2, • " " , < 1} 

is 1, for any orthogonal basis Vi, V2, ■ ■ • , of F such that 

{vi,Vi)F = ±1, i = 1,2, ■ ■ ■ ,r. 

Recall also that as a topological vector space, C~^{F) is the strong dual of the 
Prechet space of Schwartz densities on F. It contains S{F) as a dense subspace, and 
the Fourier transform extends continuously to a topological linear isomorphism 

Tf : cr^{F) ^ cr«(F), 

which is still called the Fourier transform. 

For any two closed semialgebraic subsets Zi and Z2 of F, denote by 

(3) tr«(F;Zi,Z2)c C-^F) 

the subspace consisting of all / such that 

• / is supported in Zi, and 

• ^pif) is supported in Z2. 

It is a H^Fj-submodule of C~^{F). For the rest of the section, we will be concerned 
with the structure of such PF[F]-submodules. 

For a subspace F' of F, let F'^ denote its perpendicular space: 

F'^ := G F I (v, v')f = 0, e F'}. 

Note that the Fourier transform J-F{f) of / G C^{F) is supported in F'-*- if and 
only if for all v & F' , dv is nilpotent on /. Therefore 

(4) Cr^{F; F'f^' = Cr^{F; F', F'^). 
For later use, we record the following 

Proposition 2.3. //"F° is a non- degenerate subspace of F, and 

(FY = F® © F- 
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is a decomposition to totally isotropic subspaces, then 

Cr^{F; F+ © F+ © F°) = C[F+] © CT^iF-; {0}) © Cr«(F°). 

Proof. The proof is similar to that of Lemma 12. 1[ □ 

Following |JSZj . we make the following 

Definition 2.4. (a). A suhmanifold Z of F is said to he metrically proper if for every 
z E Z , the tangent space Tz{Z) is contained in a proper non-degenerate subspace of 
F. 

(b). A locally closed subset Z of F is said to be piecewise metrically proper if there 
is a finite filtration 

Z = Zo D D ■ ■ ■ D Zfc = 

of Z by its closed subsets, so that every difference Zi \ Zj+i is a metrically proper 
submanifold of F, i = 0,1, ■ ■ ■ ,k — 1. 

Denote by the Laplacian operator on F. If vi, V2, - ■ ■ , f ^ is a basis of F, and 
v[,v'2,--- ,v'j. is the dual basis with respect to ( , )f, then 

r 

(5) Ap = J2dv^dv',. 

i=l 

The following is a special case of Lemma 3.2 in [JSZj . 

Lemma 2.5. Let F be a finite dimensional non- degenerate real quadratic space, and 
Z be a piecewise metrically proper locally closed subset of it. If f E C~^{F; Z) is 
annihilated by some positive power of Ap, then / = 0. 

Remark: A tempered generalized function / on F is annihilated by some positive 
power of Ap if and only if its Fourier transform J-'pif) is supported in the null cone 

Tp:={veF \ {v,v)p = 0}. 

Lemma 2.6. Assume dim^F = 2r. Let F^ be a totaly isotropic subspace of F 
of dimension r. Let V be a W[F]-module on which Ap and every A G {F/F^)* 
act locally nilpotently. Then every dv (v E F^) also acts locally nilpotently on V. 
Consequently, V is generated by 

{/ G V I A/ = idv)f = 0, A G {F/F^r, V G 
Proof. The second assertion follows from the first, in view of Lemma [2TT] and Lemma 

To prove the first assertion, take a totaly isotropic subspace F~ of F which is 
complementary to F'^. Note that 

W[F] = W[F+] © WIF-]. 
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View V as a -module and apply Lemma [2l2] to it, we have 

(6) V = V(g)Cr^iF-;{0}), 

where 

V = Rom^F-]{CrHF-;{0}),V) 

is canonically a H^F"*"] -module, and is an identification of iy[F]-modules. 
Take a basis Ui,U2, - ■ ■ ,Ur of F"*", and the dual basis Vi,V2, - ■ ■ ,Vr of F~ . Then 

r 

Ap = 2 ^ (9uj 

i=l 

Let /' G v. Then for some positive integer m, we have 

Af(/'®'^f-)=0, 

i.e., 

y ''i; ''2} ■ ■ ■ ; 'r y 

^1 + «2 H \-ir = m, 

ii,i2, ■■■ , V > 

® ((9wi)^i (9t;2)*' ■ ■ ■ {dvrY^ 5f- = 0. 

Therefore 

{dmY^ {dU2y'---{dUrr /' = 0, 

which proves that dui,du2, ■ ■ ■ ,dur act locally nilpotently on V', and the lemma 
follows. □ 

We are now ready to prove the following 

Proposition 2.7. Assume dim^F = 2r. Let Fi,F2,--- , Fj be a set of (distinct) 
totally isotropic suhspaces of F, each of dimension r. Then the W[F]-module 

Cr^{F; Fi U F2 U ■ ■ ■ U F„ Fi U F2 U • ■ ■ U F,) 

is completely reducible with finite length, and with each irreducible factor isomorphic 
to some (r^{F;Fi,Fi). 

Remark: We expect that 

s 

C-«(F; Fi U F2 U ■ ■ ■ U F„ Fi U F2 U ■ ■ ■ U F,) = Cr«(F; F„ F,). 

i=l 

Proposition 12.71 is nevertheless sufficient for our purpose. 
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Proof. For any nonempty open connected semialgebraic subset F° of a totaly isotropic 
subspace F"*" of F, of dimension r, set 

(7) Vf,f° := {/ G CTHF-, F1 I is nilpotent on /}. 
Then we have the restriction map 

(8) C-^iF;F+,F+)^VF,F^. 

We claim that it is a W^F] -module isomorphism. 

Clearly the map ([HD is a well-defined nonzero VF[F]-module homomorphism. It is 
injective since C~^{F; F+, F"*") is irreducible. The space Vf,f° is also a VK[F]-module 
satisfying the conditions of Lemma [2.61 Therefore, it is generated by 

{/ e Cr^{F- F°) I A/ = {dv)f = 0, A G (F/F+)*, v E F+} 

= {constant function on F°} ® 6f-, 

where F~ is a totally isotropic subspace of F which is complementary to F+. Con- 
sequently, the map ([8]) is surjective as well. 
Set 

Z :=\Jf, and Z := [j{F, n F,) . 

Label the connected components of Z \ Z by Fj^, F2°, ■ ■ ■ , F^. Clearly each of them 
is contained in some Fj as an open semialgebraic subset. Since Z is contained in 
the null cone, any / G C~^{F; Z, Z) is annihilated by some positive power of Aj?. 
Therefore the restrictions yield a VF[F]-module homomorphism 

N 

(9) Cr^iF-Z,Z)^l[VF,Fo, 

k=l 

with kernel Cr^{F;Z,Z). 
Define a filtration 

Z = Zo D Zi D ■ ■ ■ D = 

of Z by 

Zk := U {Fi n F 



dimFj nFj <r -1-k, 



i 7^ J 

Since every subspace of F of dimension < r is metrically proper, we see from the 
filtration that Z is piecewise metrically proper in F. Now Lemma [2.51 implies that 

Cr^{F;Z, Z) = 0. 

Therefore the map in (Q is injective and we finish the proof by using the isomorphism 
in m. □ 
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3. Eigenvalue estimate of an Euler vector field 

In this section, we first describe a general set-up in order to work with all five 
series of classical groups in a uniform manner. We then prove in this context an 
eigenvalue estimate of an Euler vector field acting on a certain space of tempered 
generalized functions. 

Let A he a. finite dimensional semi-simple commutative algebra over M, which is 
thus a finite product of copies of R and C. Let E be an 74-module of finite dimension, 
i.e., 

dim^(£') := max{dim^o(74o ®a E) \ Aq\s b, quotient field of ^4} < -|-oo. 
Denote by q\.a{E) the A-algebra of 74-endomorphisms of and by 

tr^ : QiA{E) A 
the trace map, which is specified by requiring that the diagram 

q{a{E) A 

QiAo{Ao®AE) Ao 

commutes for every quotient field Aq of A, where the bottom arrow is the usual trace 
map. Set 

sIa{E) := {x G qIa{E) I tr^(x) = 0}. 

From now on, we assume that a M-algebra involution r on A is given. We call {A, r) 
(or A when r is understood) a commutative involutive algebra. The commutative 
involutivc algebra A is said to be simple if it is nonzero, and has no r-stable ideal 
except for {0} and itself. Every simple commutative involutive algebra is isomorphic 
to one of the foUowings: 

(10) (M,l), (C,l), (C,-), (MxM,tm), (CxC,tc), 

where r-^ and tq arc the maps which interchange the coordinates. The first three 
cases will be referred to as Type I, and the last two cases as Type II. 
A M-bilinear map 

{,)e:ExE^A 
is called a hermitian form if it satisfies 

{u, v)e = {{v, u) eY, {au, v)e = a{u, v)e, a e A, u,v E E. 

We will always assume that E is a, hermitian A-module, namely it is equipped with 
a non-degenerate hermitian form {, )e- Denote by U{E) the group of all A-module 
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automorphisms of E which preserve the form {, )e, and by u{E) its Lie algebra, 
which consists of all x G 0U(-E') such that 

{xu, v)e + {u, xv) £ = 0, u,v E E. 

Set 

U(A) := {a G I a^a = 1}. 
Through scalar multiplication, there is a homomorphism 

U(A) ^ U(E), 

whose image, which coincides with the center of U{E), is denoted by 'Zi{E). Similarly, 

set 

u(^) := {a e A \ + a = 0}, 
and denote by }{E) the image of the map (again through scalar multiplication) 

u{A) u{E). 

Then u{A) is the Lie algebra of U{A), and i{E) is the Lie algebra of 'L{E). (Note 
that l{E) may not coincide with the center of u(i?).) Set 

su{E) := u(E) nsU(^). 

Then 

(11) U(E) =3(E)©5U(E). 



(12) 



When {A, r) is one of the five simple commutative involutive algebras in ( ITOi) . then 
accordingly, every hermitian A-module must be isomorphic to one of the followings: 

iW+'^, { , )o(p,,)), (C", ( , )oh), (Cp+^ ( , )um), 
(M"©M-, (, )m,„), (C"©C", (, )c,„), 

where p,q,n > 0, and all spaces involved are considered as spaces of column vectors. 
The corresponding hermitian forms are given as follows: ( , )o{p,g) is the symmetric 
form defined by the matrix Ipg, (, )o(n) is the standard symmetric form on C", 
( ; )v(p,q) is the usual hermitian form defined by the matrix Ip g, { , )]k „ and ( , )c,n 
are the maps given by 

flu] \ u' ]\ , ,t ,t . 
I ^ ' ^/ ] ^{v u,u v). 

The group U{E) corresponding to f|T2l) is isomorphic to one of the followings: 
(13) 0(p, g), 0„(C), U(p, g), GL„(M), GL„(C). 



(15) V'. 
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Assume in the rest of this section that A is simple. Fix an element in u{A) so 
that 

r 0, if (A,r)-(K,1), 
(14) 4=<^ -1, if (A,r) = (C,-), 

[ 1, ii{A,T) = {KxK,TK), 

where IK = M or C, as before. Note that such a ca is unique up to a sign. 
For any v & E, write 

4>y{u) := {u,v)eV, u e E, 

then (py e qIa{E). Denote by 0'^ G s[a(-E) the projection of 0„ to the second factor 
according to the decomposition 

qIa{E) = {scalar multiplication} (BsIa^E). 

For any x G su{E), set 

CA(f)'y, ifcAT^O, 
+ (f)vX, if Ca = 0, 

which is checked to be in su{E). Following [AGRSj . we define 

(16) E{x) ■.= {veE\ G [su{E),x] }. 

For any Lie subalgebra f) of QiA{E), denote by f)^ the centralizer of x in f). An 
element x G su{E) is said to be nilpotent if it is nilpotent as a M-linear operator on 
E. The following lemma gives a description of E{x). 

Lemma 3.1. Let x G su{E). 

(a) // Ca 7^ 0, then 

E{x) = {veE \ {yv, v)e = 0, y G sUiEf }. 

(b) // Ca = 0, then 

E{x) = {veE \ {xyv, v)e = 0, y G qIa{EY }. 

(c) In all cases, if x is nilpotent, then 

{x^v, v)e = for all v G E{x) and i > 0. 

Proof. We only prove Part (a). Part (b) is proved similarly, and Part (c) follows 
obviously from (a) and (b). So we assume that ca and let v E E. For simplicity, 
we write ipx,v '■= ca'P'v 'ipv 

If f G E{x), then ip^ = [z,x] for some z G 5u{E). Therefore, for all y G sIa(-E')^, 

{yv,v)E = trA(0t,l/) = trA(0'^y) 
= (ca)'^ ti AaJvy) = (caY^ tr A{zxy ~ xzy) 
= {cAy^tiAizyx - xzy) = 0. 
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On the other hand, assume that for all y G sU(i?)^, we have 

{yv, v)e = 0, i.e., tTA{ipv y) = 0. 

In particular, we have 

ip^e{z e su{E) I tiAizy) =0,ye su{Ey }. 

It is easy to see that the latter space is precisely [su(-E'),x] (c.f. [CM931 Page 14]). 
This finishes the proof. 

□ 

Denote by 

(17) TE:={veE \ {v,v)e = 0} 

the null cone of E. View ii^ as a real quadratic space by the form 

{u,v)e,r ■= tr a/r{{u,v)e), 

where tiA/R : A — > M is the usual trace map for commutative algebras. 

For any finite dimensional real vector space F and any x G EndiR(F), denote by 

(18) ep,, G W[F] 

the vector field on F whose tangent vector at f G -F is . When x = 1 is the identity 
operator, this is the usual Euler vector field ep ■= €f,i. 
For a nilpotent element e G su{E), define 

(19) Vs,e := C-^iE; E{e) n F^,, E{e) n Fij)^^^), 

where and as usual, a superscript group indicates the group invariants. Clearly 

(20) eE,cJ = 0, / G VE,e. 

The space V^.e arises naturally when one carries out the reduction within the null 
cone. See Lemma [4.71 

Proposition 3.2. Assume that dimA{E) = 1. 

(a) If A is of Type I, then 

cr^{E;rE,rE) = {o}. 

(b) If A is of Type II, then for every f G C^^{E; Tp, ^e), 

(^E,CAf = implies / = 0. 
Consequently, in all cases, Ve^b = {0} for the only element e G 5u{E) = {0}. 
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Proof. In case (a), we have Te = {0}, which is metrically proper in E. Therefore 
the lemma follows from Lemma 12. 5[ 
In case (b), we assume that 

{E,{,)e) = {K®K, 

as in f|T2|) . Then 

r^; = Fo U Fi 

is the union of two totally isotropic subspaces Fo and Fi, where 

Fo:={0}©K, and Fi:=K©{0}. 

By Proposition \2.7\ it suffices to show that for every / G C^^{E; F^, Fi), 

(^E,CAf = implies / = 0. 

To fix the sign, assume that = (1, —1). By Lemma [2.H 

Cr^{E; Fo, Fo) = C[Fo] ® CT^iF,; {0}), 

and therefore eE,CA ci-cts semisimply on it, and all its eigenvalues are negative integers. 
Likewise, eE,CA ^^cts semisimply on C~^{E; Fi, Fi), and all its eigenvalues are positive 
integers. This finishes the proof. □ 

Recall that a nilpotent element e G su(F) is said to be distinguished if it commutes 
with no nonzero semisimple element in su(F) (c.f. |CM93l Section 8.2]). 
The rest of this section is devoted to a proof of the following 

Proposition 3.3. Let h, e, f G su(F) be a standard triple, i.e., 

[h, e] = 2e, [h, f] = 2f , [e, f] = h. 

Assume that dim^(F) > 2 and e is distinguished. Then the vector field tE;h acts 
semisimply on Ve,^, and all its eigenvalues are nonnegative integers. 

For every h G su(F) in a standard triple h, e, f , denote by F^ C E the eigenspace 
of h with eigenvalue i, where i G Z. Write 

^h:=0^h, and E^:=@El 

i>0 i<0 

Next we prove (a stronger version of) Proposition 13.31 when A is of Type I. 

Lemma 3.4. Assume that A is of Type I and dim^(F) > 2. Let h, e, f G su(F) be 

a standard triple, where e is a distinguished nilpotent element in su(F) . Then 

(a) F(e) is contained in E^ + E^; 

(b) the vector field eE,h acts semisimply on C^^{E; F(e), F(e)), and all its eigen- 
values are nonnegative integers. 
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Proof. As usual, view E as a sl2(K) ®r A-modules via the standard triple. Let 
(21) E = Ei®E2®---®E, 

be a decomposition of E into irreducible s[2(IR) ®r A-modules. By the classification 
of distinguished nilpotent orbits ( |CM93l Theorem 8.2.14]), we know that s = 1 if 
{A,t) = (C, "). If {A,t) = (K, 1), then (!2T!) is an orthogonal decomposition, and 
Ei,E2, - ■ ■ ,Es have pairwise different odd dimensions. 

Suppose that we are in the latter (orthogonal) case. Denote by e,; G 5u{Ei) the 
restriction of e to Ei. It is easy to see that ([ AG RSI Lemma 5.3]) 

E{e) C Ei(ei) + ^2(62) + ■ • • + Erier). 

To show Part (a), we may therefore assume that E is irreducible as a s[2(M) (8>r^- 
module. Let r > be its highest weight and 

{vi \ i = —r, — r + 2, ■ ■ ■ , r} 

be an A-basis of E such that 

• Vi is an eigenvector of h with eigenvalue i, and 

• evi = Vi+2, i < r. 
Assume that there is an element 

V = a_rV-r + a_r+2V-r+2 H h O-rVr ^ E{e) \ {E^ + E^). 

Denote by j > the largest number so that a_j 7^ 0. Then 

(e%, v)e = a^j {vj, v^j)E 7^ 0, 

which contradicts Part (c) of Lemma 13.11 This proves Part (a) . 
By Part (a) and Proposition 12.31 we have 

Cr^{E;Eie),Eie)) 
C Cr^E; E^ + El E^ + E'^) 
= C[Ei](g>(rHE^;{0})®(r^iEl. 
Part (b) therefore follows. □ 



We are now left with the task of proving Proposition 13. 3[ when A is of Type II. 
Thus let {A, r) = (K X K, tk), and 

(E,(,)s) = (K"©K",(,)k,„) 

be as in ffT2l). with n> 2. Then 



U(E) 



9 

g-' 



g E GL„(K) \ = GL„(K) 
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and 



u{E) 



X 
-X* 



X e 0l„(K) } = g[„(K) 



A distinguished nilpotent element e of su{E) is principle and we may assume 
without loss of generality that 






1 


■ 


■• 











1 ■ 


■■ 











■ 


■■ 


1 








• 


•• 






and 

h = diag(n — 1, n — 3, ■ ■ ■ , 3 — n, 1 — n). 
Then it is easy to check that 

E{e) n r^, = Fo U Fi U ■ ■ ■ U 

where 

Fi = {W © {0}"-^ © ({0}' © W-'). 
In view of Proposition 12.71 and ( l20l) . it suffices to prove the following 

Lemma 3.5. With notation as above, the vector field eE,h o,cts semisimply on 

(22) {fEC-^iE;F,,F,)\eE,cJ = 0}, 

with all its eigenvalues nonnegative integers. 

Proof. We prove the lemma for K = M. The complex case is proved in the same way. 

Denote by Xi, X2, ■ " " ? 3;„, yi,y2, - ■ ■ ,yn the standard coordinates of © M", and 
write 

d d 
dj = - — and dj = — — , j = 1, 2, 



dxj dyj 



,n. 



By Lemma [^TT| the space Cr^{E] F,, Fi) has a basis consisting of generalized func- 
tions of the form 



J — Xi X2 Xi y^j^i (/j_,_2 y„ 

®CV"'C2'"' ■ ■ ■ d'r'd'r' ■ ■ ■ d^-'dp;, 

where ai, Oj, fej+i, 6„ are nonnegative integers, and the rest of a's and 6's are 
positive integers. Here and as before, 6f' is a fixed Dirac function on the space 



:= ({0}' © K"-') © {W © {0}^ 
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The generalized function / as above is an eigenvector for both eE^CA ^E,h- The 
condition 

eE,CAf = 

amounts to 

Y^{aj + hj) = ^(aj + bj). 

j<i j>i 

Then the eE,h-eigenvalue of / is 

{n — l)ai + {n — 3)a2 + ■ ■ ■ + {n — 2i + l)ai 

— {n — 2i — l)aj_|_i — {n — 2i — 3)aj_|_2 — ■ ■ • — (1 — n)an 
+ {n - l)bi + (n - 3)62 + ■ • ■ + (n - 2i + l)bi 

-{n - 2i - - {n-2i- 3)6j+2 (1 - n)bn 

>{n — 2i)ai + {n — 2i)a2 + ■■■ + (« — 2i)ai 

— {n — 2i)ai+i — {n — 2i)aj+2 — ■ ■ ■ — (n — 2i)a„ 

+ (n - 22)61 + {n- 2i)b2 H V {n - 2i)bi 

-{n- 2i)bi+i - {n- 2i)bi+2 - ■ ■ • - (n - 2i)bn 
= 0. 

Note that for i = 0,n, the space is equal to zero. 

4. Reduction within the null cone 

Recall that we are given a commutative involutive algebra {A, r) and a hermitian 
A-module E. Denote by U{E) the subgroup of GL]k(_E') x {±1} consisting of pairs 
{g, 5) such that either 

6=1 and g e \J{E), 



□ 



or 



(23) 



Denote by 
(24) 



6 = -l, 

g{av) = a^g{v), a E A, v E E, and 
{gu, gv)E = {v, u)e, u,v e E. 



XE : tJiE) ^ {±1} 



the quadratic character of U{E) projecting to the second factor, which is easily 
checked to be surjective. Therefore, we get an exact sequence 

{1}-.V{E)^1]{E)'^{±1}^{1}. 
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Now let U(E) act on \J{E) by 

(25) {g,6)x:=gx'g-\ 
and on E by 

{g,6)v := 5gv. 

Let IJ(-E') act on u{E) through the differential at the identity of U(£'), i.e., 

{g,6)x := 6gxg~^. 

Let IJ{E) act on V{E) x E and u{E) x E diagonally. 

We introduce the following general notation. If H is a Lie group acting smoothly 
on a manifold M, then for any character xh of H, denote by 

(26) CTni^^) ^ cr^{M) 

the subspace consisting of all / which are x^-equivariant, i.e., 

f{hx) = Xnih) f {x) , for all h E H. 

Similar notations (such as C^^{M; Z)) apply without further explanation. We will 
be concerned with the space C^^{u{E) x E). 

Denote by N'e C su{E) the null cone, which consists of all nilpotent elements in 
su{E). Let 

= ATo D M D ■ ■ ■ D A4 = {0} D A4+1 = 
be a filtration of JVe by its closed subsets so that each difference 

is an U(-E')-adjoint orbit. 

Our aim is to prove the following reduction result for C^|(u(i?) x E). Recall the 
null cone F^; of E. 

Proposition 4.1. Assume that A is simple, dimA{E) > 1, and that every element 
of C^^{u{E) X E) is supported in {i{E) + Afi) x F^;, for some fixed < i < k. Then 
every element of C^^{u{E) x E) is supported in {]){E) + A/i+i) x F^;. 

Note that u{E) = i{E) ® 5Vi{E) is a IJ(-E) stable decomposition, and IJ(-E) acts 
on ]j{E) trivially. Therefore by the localization principle (See |JSZ[ Lemma 4.1], for 
example), for any fixed z, 

every element of C^|(u(-E') x E) is supported in {i{E) + Mi) x Fg 

<^=^ every element of C^|(su(i?) x E) is supported in Mi^Te- 

Thus it suffices to prove the following equivalent 
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Proposition 4.2. Assume that A is simple, dim^(£') > 1, and that every element 
of C^^{su{E) X E) is supported in Mi x V e, for some fixed < i < k. Then every 
element of C^^{su{E) x E) is supported in A/^+i x F^;. 

For the ease of notation, denote 

s := su{E). 

We shall view s as a non-degenerate real quadratic space via the form 

{x,y)s,R ■= tiA/mitTAixy)). 

Note that the null cone Me is contained in the null cone of s as a real quadratic 
space. 

Lemma 4.3. Let O C Me be a nilpotent \J{E)-orbit which is not distinguished. 
Then O is metrically proper in s. 

Proof. Let x E O. By definition, it commutes with a nonzero semisimple element 
h E s. Denote by at the center of s'*, which is a nonzero non-degenerate subspace of 

5. 

Using the fact that every element of at commutes with x, we see that the tangent 
space 

T,{0) = [u{E),x] 
is contained in the proper non-degenerate subspace 

(ah)^ := {y esl {y, z)^^^ = 0, z e Uh} c s. 

□ 

Lemma 4.4. Proposition^^ holds when Oi is not distinguished. 

Proof. Let / G C^|(s x E). Then J^^^f) G C^|(5 x E), where JF^ is the partial Fourier 
transform (along s) specified by the commutative diagram 

c^«(s X E) cr^{s)®cr^{E) 

cr^isxE) cr^{s)^cr^{E). 

By the assumption, the support of J-'s{f) is contained in 

Mi X Te CI (the null cone of the real quadratic space s) x E. 

Therefore, some positive power of the partial Laplacian Ag annihilates /. Now the 
lemma follows from Lemma [4.31 and (a variation of) Lemma [2.51 □ 
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Before proceeding further, we introduce a version of pull back of generalized func- 
tions. 

Definition 4.5. Let Z and Z' be locally closed subsets of manifolds M and M' , 
respectively. A smooth map (p : M M' is said to be submersive from Z to Z' if 

• (f) is submersive at every point of Z , and 

• for every z & Z , there is an open neighborhood U of z in M such that 

(l)-\z')r]U = z nu. 

The following lemma is elementary. 

Lemma 4.6. If (p : M ^ M' is submersive from Z to Z' , as in Definition \4.5\ then 
there is a unique linear map 

(27) 0* : C^°°(M'; Z') C^°°(M; Z), 

with the following property: for any open subset U of M and U' of M' , if 

• (j) restricts to a submersive map 4>u '■ U ^ U' , 

• Z' nU' is closed in U' , and 

then the diagram 



^\z'r]U') = zr]U, 



cr^{M'-z') cr^{M-z) 



commutes, where the two vertical arrows are restrictions, and the bottom arrow is 
the usual pull back map of generalized functions via a submersion. 

See |W88l Lemma 8. A. 2. 5] for the definition and properties of the usual pull back 
map. Note that the vertical arrows are well defined since Z' fl U' is closed in U', and 
Z nU is closed in U. The map 0* in (l27j) is still called the pull back. It is injective 
if (piZ) = Z'. In this case, we say that (p is submersive from Z onto Z'. 

We continue the proof of Proposition 14.21 Recall the notations from Section [31 

Lemma 4.7. Under the assumption of Proposition the support of every f e 
C^|(s X E) is contained in 



(M+i X r^) u ( □ {e} X (E(e) n r^)). 
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Proof. We follow the method of [AGRSj . For every t G M, define a map 

rj := rjt : 5 X E s x E, 



which is checked to be submersive from sxTe to sxTe- Therefore, by Lemma [4. 6[ 
it yields a pull back map 

7]* : CT'^isx E;5xTe) ^ Cr°°{5 x E;s x Te). 

Fix / G C^|(s X E). By our assumption, its support is contained in Afi xTe, and 

so 

/ G q«(5 X X r^;) C C^lis X E-5 X Te). 

Since the map rj is algebraic and U(-E')-equivariant, 

r/*(/)G Cr^l{5xE-5xVE). 

It is routine to check that rj restricts to a bijection from s x F^; onto itself. Let 
(e, t>) G Cj X F^; be a point in the support of /. Denote by 

e' := e'(e, v,t) G s 

the unique element so that 

r]{e\v) = {e,v). 

Then (e',f) is in the support of ri*{f), and therefore our assumption implies that 

(28) e' G TVi- 

A easy calculation shows that 

e + ttpe,v, if 7^ 0, 

e + t'ijje,v + t^(l)v^4>v, if Ca = 0. 

Since Oj is open in A/^, fl28|) implies that 

i^e,v = j^\t=oe'{e,v,t) G T,{0,) = [u(^),e] = [su(E),e], 
i.e., V G E{e), and the proof is now complete. □ 

For a nilpotent U(i?)-orbit O C Me, denote by 

(29) Vaxi?,o cCr^{sxE;Ox E)U(^) 

the subspace consisting of all / such that the supports of both / and its partial 
Fourier transform J-'eU) ^-^^ contained in Ueeo^^} ^ (-^(^) ^e)- 



e' 
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Lemma 4.8. Assume that A is simple, dim^(i?) > 1, and O is distinguished. Then 
the Euler vector field acts semisimply on Vsxe,0! ^.''^d all its eigenvalues are real 
numbers < — ^diniKS. 

Let us prove the following 

Lemma 4.9. Lemma \4-^ implies Proposition ^.^ when Oi is distinguished. 



Proof. Denote by the quadratic form on s, i.e., 

gs(x) = (x,x)s,K = tr^/iR(trA(x^)). 

The operators 

es + -dimK3, --q,, -A^ 

form a standard triple in W[s\, and each of them leaves the space Vsxe,o, stable. 
Lemma 14.81 says that + | dimig s is semisimple and has negative eigenvalues on 
VsxE.c, and so by |W88l Lemma 8. A. 5.1 ], the map 

is injective. 

Let / G {s X E). Applying Lemma 14.71 to / and its partial Fourier transform 
J-'sif), we conclude that under the restriction map 

r^xE : X ^ Cr^{5 xE-0,x E), 

the image 

rsxEif) e VsxEA- 

Since the partial Fourier transform J-'s{f) is again in C^|(s ^ E), we see that J^sif) 
is supported in 

J\fi X Te C (the null cone of the real quadratic space s) x E, 

which implies that r^y^Eif) is annihilated by some positive power of Ag. By the 
injectivity of A^ on VsxE,Oi, we conclude that r^xEif) = and we are done. □ 

The remaining part of this section is devoted to the proof of Lemma 14.81 

For the moment, assume that O C Me is any nilpotent U(i?)-orbit (not necessarily 

distinguished). Pick any element e G (9 and extend it to a standard triple h, e, f G s. 

Then we have a vector space decomposition 

5 = [s, e] © 5^. 

Let U{E) act on U(-E') x x E via the left translation on the first factor. Define 
a U(i?)-equivariant map 

. . e : \J{E) xs^ X E sx E, 

{g,x,v) ^ g{x + e,v). 
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Lemma 4.10. The vector field 
(31) 

on U(i?) X 5^ X E is 6-related to the Euler vector field Cg on s x E, where th/2 is the 
left invariant vector field on U(i?) whose tangent vector at the identity is h/2. 

Proof. Since both vector fields under consideration are U(ii^)-invariant, it suffices to 
prove the 6'-relatedness at a point of the form 

X := {l,x,v) e U(E) X5^ X E. 

Applying the differential of 9 at x, we have 

^h/2|x = (h/2,0,0) ^ ([h/2,a; + e],(h/2)t;), 

esM-ad(h/2)|x = (0,x- [h/2,x],0) ^ (x - [h/2, x], 0), 
ei?,h/2|x = (0,0,(h/2)t;) ^ (0, (h/2)t;). 

This implies the lemma since eg|0(x) = (x + e, 0). □ 

Let Z(ii^) act ons^ x E and U(-E) xs^ x E via its action on the factor E. Then the 
map 9 is Z(i5)-equivariant as well. Note that 9 is submersive from U(i?) x {0} x E 
onto O X E (c.f. |W88t Page 299]). Therefore it yields an injective pull back map 

Cr^{5 xE;Ox E)U(^) 
^ Cr^{\J{E) X / X V{E) X {0} x ^)U(i?)xZ{i?)_ 



Denote by 

C^«(U(E) X / X E;\J{E) X {0} x ^)U(s)xZ(b) 
C-«(/ X ^; {0} X E)"(^) 

the linear isomorphism specified by the rule 

/ = 1 OrgfxB/. 

Write Vgfx_B,e for the space 

cr^is' X E- {0} X (E(e) n r^), {o} x {E{e) n r^))"(^). 

In previous notations (see ([T]) and ffT^ ). we have 

(32) V,.^E,.= C-^{5'-m®VE,.. 
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Lemma 4.11. The composition map r^t^^ ° ^* sends Vsxe,o ii^io V^fxE.e; o^^^ the 
following diagram 

VsxE,0 " ^ Vgf xE.e 

Sf,l-ad(h/2)~^-B,h/2 



commutes. 



Proof. The first assertion follows by noting tliat botli 6* and r^t E commute witli 
tlie partial Fourier transform along E. Tlie second assertion follows from Lemma 

Km □ 

Lemma 4.12. Assume that dimA(-E') > 2. Then the vector field egf i_ad(h/2) o,cts 
semisimply on C~^(s^; {0}), and all its eigenvalues are real numbers < — |dim]RS. 

Proof. The condition dim^(£') > 2 implies that s ^ {0}. 

We view s as a sl2(K)-niodule via the adjoint representation and the standard 
triple {h, e, f}. We shall prove that the analog of Lemma [4.121 holds for any finite 
dimensional nonzero s[2(M)-module F. Without loss of generality, we may assume 
that F is irreducible of real dimension r + 1. Then 

eFf,i-h/2 = {^+r/2)epf, 

which clearly acts semisimply on G~^{F^; {0}), with all its eigenvalues real numbers 
< -(l+r/2) = -idimsF- i < -idimKF. □ 

In view of ( l32l) , Lemma 14.81 will follow from Lemma 14. IH Lemma 14.121 together 
with Propositions 13.21 and 13.31 

5. Reduction to the null cone 
We first recall the following elementary (and well-known) lemma. 

Lemma 5.1. Let H be a Lie group acting smoothly on a manifold M. Let xh be 
a continuous character on H . If C^^{M) = 0, then C^^{M') = for any open 
submanifold M' of the form (f)~^{N'), where (p : M ^ N is an H-equivariant smooth 
map, N is a manifold with trivial H-action, and N' is an open submanifold of N. 

Recall that {A, r) is a commutative involutive algebra, and ii^ is a hermitian A- 
module, as well as other notations from Section [3l The following result may be 
considered of Harish- Chandra descent. 
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Proposition 5.2. Assume that A is simple, dim^(i?) > 1, and for all commutative 
involutive algebra A' and all hermitian A' -module E' , 

(33) <lmvA'{E') <<limA{E) implies q^^(U(E') x E') = 0. 

Then every f E C^'^{U{E) x E) is supported in {Z{E)IAe) x E, where Ue is the set 
of unipotent elements in U{E). 

Proof. Extend the involution r on A to qIa^E) (still denoted by r), by requiring that 

{xu,v)e = {u,x'^v)e, X E glA{E), u,v E E. 

Now let a; be a semisimple element in U(i?) \ 'Zi{E). Let A' be the M-subalgebra 
of qIa{E) generated by A, x and x'^ , which is a commutative involutive algebra. Put 
E' = E, but viewed as an A'-module. Define a map 

{,)e':E'xE'^A' 

by requiring that 

tYA'/R{a{u, v)e') = tTA/R{{au, v)e), aE a', u,v E E. 

Then E' becomes a hermitian A'-module, with 

dimA'{E') < dim^(E), 

and U(-E') coincides with the subgroup of IJ{E) consisting of all {g, 6) such that 

-1 fx, if 5 = 1, 
=\x^ if5 = -l. 

For any y E 1]{E'), denote by J{y) the determinant of the M-linear map 

1 - Ady-i : u{E)/u{E') u{E)/u{E'). 

Note that Adj^ preserves a non-degenerate real quadratic form on u{E)/u{E'). This 
implies that J is a U(i?') -invariant function (under the action (l25l) ). Put 

UiE'y ■■= {y G U(E') I J{y) ^ 0}, 

which contains xUe'- The map 

TT : IJ(E) X (U(E')° X ^ U(^) X E, 

is a submersion. Therefore we have a well defined restriction map ( |JSZt Lemma 
4.4]) 

rE,E' : C-^Ti^(E) X E) ^ q^^(U(E')° x E'), 
which is specified by the rule 

7r*(/) = XE^rE,E'{f). 
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The assumption (l33ll and Lemma [5.11 imply that the later space is zero. Thus every 
/ G C^^(U(-E) X E) vanishes on the image of vr. As x is arbitrary, the proposition 
follows. □ 

The remaining part of this section is devoted to the proof of the following 

Proposition 5.3. Assume that A is simple, dimA{E) > 1, and for all commutative 
involutive algebra A' and all hermitian A' -module E' , 

(34) dimA/(E') < dimA(E) implies C^lX^{E') x E') = 0. 

Then every f G C^l{u{E) x E) is supported in {^{E) + JVe) x Te- 

Similar to the proof of Proposition 15. 2^ we show that every / G {''^{E) x E) 
is supported in {}{E) + Me) x E. We are left to show that / is also supported in 
n{E) X Te. 

Fix 

t G := {a G I a" = a}, 

and set 

E{t) ■.= {veE \ {v,v)E = t}. 
Fix vq G E{t) (when it is nonempty), and put 

E' ■.= {veE \ {v,vo)e = 0}. 

Then 

E = E'® Avo 

is an orthogonal decomposition of hermitian A-modules. We identify U(-E") with a 
subgroup of U{E) via the embedding 



(35) ig,5) 



5g 

Ts 



6 



where : Avq — >■ Avq is the M-linear map given by 

avo, if 5 = 1, 



Ts{avo) 



— a^t>o, if 6 = — 1. 



Then U{E') is precisely the stabilizer of vq in U(i?). 
Let U{E') act on u{E) by way of the action of U{E). 

Lemma 5.4. With notations as above, we have 

u{E) = u{E') xE' X u{Avo) 

as R-linear representations ofU{E'). Here U(£") acts on u{Avq) trivially. 
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Proof. Denote by pe' the projection to the first factor according to the orthogonal 
decomposition 

E = E' ® Avq. 

For any 



X 

set 



X21 X22 



e u{E), 



■Ki{x) := Xn, TT2{x) := Pe'{xvo), and iTsix) := X22- 
It is routine to check that the map 

TTi X X TTs : u{E) Vi{E') x E' x u{Avq) 

is injective and U(£")-intertwining. By comparing the dimension, we see that the 
map is an isomorphism. 

□ 

We now finish the proof of Proposition 15.31 Define a map 

: X u(E) X ^ u{E)xE, 

{a,x,v) {x,av). 

Let 1]{E) act on (A^)^i'^> trivially. Note that \J{E) preserves E(t), and is transitive 
on E[t) by Witt's lemma. The map pt is a U(i?)-equivariant submersion, and hence 
defines a pull back map 

pl : C;i{u{E) X E) C^i((AX)ii-> X u{E) x E{t)). 

We have 

q|((AX)^i'-}xu(E)xE(t)) = o 

'^=> C^|((u(£') X E{t)) = the localization principle 
<^=^ C^|, (u(-E')) = Frobenius reciprocity 

Crli^uiE') X E' X u{Avo)) = Lemma EH 
<^=^ C^^^{u{E') X E') = 0. the localization principle 

For a proof of Frobenius reciprocity, see [AG 2 1 Theorem C.3.3] for example. As the 
last equality holds by assumption ( IMI) . the first equality holds. Consequently, every 
/ G C^^{u{E) X E) vanishes on the image of p^. As t G {A'^)^^''^^ is arbitrary, and 
we conclude that / is supported in \x{E) xVe- 
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6. Proof of Theorem [B] 

Let {A, r) be a commutative involutive algebra, and let ii^ be a hermitian A- 
module, as in Section [3l Recall the group U{E) and the quadratic character xe of 
\]{E), as in ([231) and (El]). 

Lemma 6.1. Write 

A = Ai X A2 X ■ ■ ■ X Ar 

as a product of simple commutative involutive algebras. Set 

Ej = Aj E, 
which is canonically an hermitian Aj -module. 
(a) // 

(36) a,^iV{E,)xE,) = for all t = 1,2,- ■■ ,r, 



then 
(b) // 
then 



C^^{\J{E)xE) = 0. 
Cr^lXuiEi) X E,) = for all I = 1,2, 



C^i{n{E)xE) = 0. 
Proof. Let us prove Part (a). Part (b) is proved similarly. Note that 

E = El X E2 X ■ ■ ■ X Er, 

and 

U(^) = V{Ei) X V{E2) X ■ ■ ■ X \J{Er). 

Recall the following elementary fact (c.f. |AGSll Proposition 3.L5]). Let Hi be 
a Lie group acting smoothly on a manifold Mi, and let H'^ be a subgroup of Hi, 

Cr°^{Mif'^ = CT'^iMi)^^ for alH = 1, 2, ■ • • , r, 



i = l,2,--- ,r. If 



then 

^H[xH^x---xH^ 
\HixH2X---xH, 



(37) C^~(Mi X M2 X ■ ■ ■ X Mfc)^^i-"2' 



= C^°°(Mi X M2 X ■■■ X MkY 
Note that is equivalent to 

C^°^(U(Ei) X Ei)^^^'^ = C^°°(U(E,) X E,f^^'\ 
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By (l37|l . we have 

C'^{\J{E) X _£;)U(Si)xu(£;2)x-xu(£;r) 
= (7~°°(U(-E') X _£;)U(Si)xu(£;2)x-xu(£;o_ 

Now Part (a) of the lemma follows by noting that, as operators on U(£') x E, the 
group U{E) coincides with the subgroup of U(-E'i) x \J{E2) x ■ ■ ■ x lJ{Er) consisting 
of elements of the form 

□ 

Lemma 6.2. Let H be a reductive linear algebraic group defined over M, with an 
algebraic action on a finite dimensional real vector space F. Let xu be a (continuous) 
quadratic character ci/H(]R). Then 

C^^{F) = if and only if = 0. 

See |AG2l Theorem 4.0.8] for a proof, which uses geometry invariant theory. 

Proposition 6.3. One has that 

q^^{u{E)xE) = 0. 

Proof. By Lemma 16.21 we only need to prove that 

(38) q|(u(E)xE) = 0. 

We prove fl38l) by induction on dim^(i?). When dim^(i?) = 0, we have U{E) = 
{±1} and so fl38l) is trivially true. So assume that dim^(£') > 1, and that fl38l) 
holds for all commutative involutive algebra A' and all hermitian A'-module E' with 
dimA'(^') < dim a{E). 

By Lemma 16.11 we may further assume that A is simple. By Proposition 15. 3[ we 
see that every / e C^|(u(-E) x E) is supported in {^{E) +Af) xTe- Proposition 14.11 
then implies that / = 0. □ 

Proposition 6.4. One has that 

C^^{\J{E)xE) = 0. 

Proof. Again, we prove by induction on dimA{E). When dimA{E) = 0, the propo- 
sition is trivially true. So assume that dimA{E) > 1, and that the proposition 
holds for all commutative involutive algebra A' and all hermitian A'-module E' with 
dimA'(^') < dimA(^). 

By Lemma 16.11 we may further assume that A is simple. By Proposition 15. 2[ we 
see that every / G Cr^{U{E) x E) is supported in {Z{E)Ue) x E. 
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Define a U(i^^)-equivariant map 

Pe : Z(E) X 5\x{E) X E ^ \J{E) x E, 
{z,x,v) (zexp{x),v). 

As is well-known, pE is submersive from Z{E) xMe x E onto {Z{E)Ue) x ii^. Therefore 
it yields an injective pull back map 

C^^{\J{E)xE; {Z{E)Ue)xE) 

^ Cr^{Z{E) X su{E) X E; Z{E) x ATe x E). 



To finish the proof, it suffices to show that 

q~(z(E)x5u(E)x£;) = o. 

Since U(i?) acts on Z{E) trivially, by the localization principle, this is equivalent to 

C^'^{su{E)xE) = Q. 

Again, since IJ(-E) acts on i{E) trivially and u(£') = l{E) ©su(i?), this is equivalent 
to 

C;^-(u(E)xE) = 0, 

which is asserted by Proposition 16.31 □ 

Theorem 6.5. Let E he a hermitian A-module and assume that A is simple. Let 
Vq E E \ Te, and define 

E':={veE \ {v,Vo)e = 0}, 

which is a hermitian A-module. Identify U{E') with the stabilizer of vq in U{E) via 
the embedding and let U(i?') act on \J{E) by way of the action ofU{E). Then 

q-(u(E)) = o. 

Proof. As in Section [5l put 

t:= {vo,Vo)e{A-)^''^\ 

and 

E(t) ■.= {veE \ {v,v)E = t}, 

which is an U(£')-homogeneous space. 

Fix a U(-E)-invariant positive measure pE{t) on E{t), and a Lebesgue measure pe 
on E. Define a map 

Jt : cr°°iE{t)) cr^{E) 
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by requiring that the diagram 



C^°°(E(t)) 



Jt 



commutes, where D~°° stands for the space of distributions, the lower horizontal 
arrow is the push forward of distributions via the closed embedding E{t) ^ E, and 
the vertical arrows are linear isomorphisms given by multiplications of the indicated 
measures. 

Then we have an injective continuous linear map 



The later space vanishes by Proposition 16.41 and therefore so does the former one. 
We finish the proof by using Frobenius reciprocity. 



We now show that Theorem [B] is implied by Theorem 16. 5[ Let A be one of the five 
simple commutative involutive algebras as in ffTOl) . and E be the hermitian A- module 
as in (|T2|) . with n,q > 1. Let Vq be the vector in \ F^; given by 



\ [0,0, ■■■ ,0,1,0,0,- ■■ ,0,1]*, if A is of type IF 

Then G' of Theorem [B] coincides with U{E') of Theorem l6.5[ Define ctq := {—Xq, — 1), 
where xq G GL]R(ii^) is given by 



Then cxq is an element of IJ(E) \ V{E) fixing vq, and so is in IJ(E') \ V{E'). See d^SD 
for the description of U(-E) and ( l35l) for the explicit embedding of U(i?') in U(-E). 
Theorem [B] follows from Theorem 16.51 by observing that ctq yields the anti-involution 
a of G = U(-E), as desired. 



1 ® J* : (U(^) X E{t)) ^ C^^{\J{E) X E). 



□ 




if A is of type I, 




7. Theorem [B] implies Theorem \M 



This section is devoted to a proof of the following proposition, which says that 
Theorem [B] implies Theorem |X1 in a general setting. The argument is standard. 
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Proposition 7.1. Let G be a real reductive group, with a reductive closed subgroup 
G' . Let a be a continuous anti- automorphism of G which leaves G' stable. Assume 
that for every generalized function f on G, the condition 

figxg'^) = fix) for all g eG' 

implies 

fi^n = /(^)- 

Then for all irreducible Harish- Chandra smooth representation V and V of G and 
G' , respectively, we have 

dim Home (V',^^') < 1- 

We will use the following form of Gelfand-Kazhdan criteria. A more general version 
is proved in [SZ, Theorem 2.2]. 

Lemma 7.2. Let S be a closed subgroup of a real reductive group H, and let a be a 
continuous anti- automorphism of H . Assume that every bi S-invariant generalized 
function on H is a-invariant. Then for any two irreducible Harish- Chandra smooth 
representations Uh and Vh of H which are contragredient to each other, one has that 

dimHoms(t/i^,C) dimHom5(yf^, C) < 1. 

We return to the proof of Proposition 17. 1[ Set 

H:=Gx G', 

which contains G as a subgroup. Denote hj S C H the group G' diagonally embedded 
in H. For any x = [g, g') G H, set 

:= ig-^r)- 



Lemma 7.3. Under the assumption of Proposition 7.1, if f is a bi S-invariant 
generalized function on H , then 

f{xn = fix). 

Proof. The multiplication map 

tuh: SxGxS H 

isug,S2) sigs2 

is a surjective submersion. Let / be a bi ^-invariant generalized function on H. 
Then its pull back has the form 

m^(/) = l®/G®l, with /g G C-°°(G). 
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By considering the commutative diagram 



SxGx S > H 

rriH 



Ads X Adj, X Adj, 



Ad, 



SxGx S > H, 



for all s e 5*, we conclude that fc is invariant under the adjoint action of G' . By the 
assumption of Proposition 17.11 we conclude that fc is a-invariant. 
Set 

(si, ^, S2Y := (4, g", s"^), {sug, s2)eSxGx S. 

Then 1 ® /g ® 1 G C^°^{S x G x S) is also a-invariant. We conclude that / is 
(T-invariant by appealing to the commutative diagram 

SxGx S > H 

rriH 



SxGx S > H. 

□ 

Lemma 7.4. We are under the assumption of Proposition \n\ Let {Vh,p) be an 



irreducible Harish- Chandra smooth representation of H . 

(a) Set 

■■= pih-n- 

Then {VH,P-a) is an irreducible Harish- Chandra smooth representation of H 
which is contragredient to {Vh,p)- 

(b) We have 

dimHom5(Vff,C) < 1. 

Proof. Denote by 

x,eCr^{H) 

the character of {Vh, p)- Then its contragredient representation has character Xp{h~^)- 
It is clear that {Vh, P-a) is an irreducible Harish-Chandra smooth representation, 
with character xp{h~"). Note that the assumption of Proposition 17.11 easily implies 
that every generalized function on H is a-invariant provided it is invariant under the 
adjoint action of H. Since a character is always invariant under the adjoint action, 
we conclude that 

Part (a) then follows from the well-known fact that an irreducible Harish-Chandra 
smooth representation is determined by its character. 
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For Part (b), denote by Uh the irreducible Harish- Chandra smooth representation 
which is contragredient to Vh- Lemma [7.21 and Lemma [7.31 imply that 

dimHoms(f/i^,C) dimHom5(VH, C) < 1. 

Now Part (a) clearly implies that 

dim Hom5'(f///, C) = dimHom5(VH, C). 

We therefore conclude that dimHom5(VH, C) < 1. □ 

We now finish the proof of Proposition 17.11 Denote by U' the irreducible Harish- 
Chandra smooth representation of G' which is contragredient to V. Set 

Vh := Vm', 

which is an irreducible Harish-Chandra smooth representation of H. As usual, we 
have an obvious linear embedding 

HomG'(V^, V) ^ RomsiVn, C). 
The later space is at most one dimensional by Lemma 17. 4[ and so is the former. 
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